Comment on Bachmann et al. (2013): A nonrepresentative sample cannot describe the extent of cultural eutrophication of natural lakes in the United States

In their recent paper, Bachmann et al. (2013) evaluate the extent to which natural lakes in the contiguous United States have been affected by cultural eutrophication since European settlement, using paleolimnological data collected during the 2007 National Lakes Assessment (NLA; U.S. Environmental Protection Agency [USEPA] 2009). The NLA sites were selected using a statistically valid sampling design that allows for the overall ecological condition of the nation’s lakes to be accurately characterized (USEPA 2009). Given the current consensus among limnologists regarding the prevalence of culturally eutrophic lakes (Finlayson and D’Cruz 2005; Carpenter et al. 2011), the conclusion of Bachmann et al. that ‘‘in the United States of America, the extent that natural lakes have been changed by cultural eutrophication does not seem to be large’’ (Bachmann et al. 2013, p. 950) is surprising. The findings of Bachmann et al. supporting this statement are not based on the entire NLA sample of natural lakes but rather on a subset of them. We demonstrate below that not only is this subset not representative of the entire population of natural lakes in the United States, but that it is biased toward lakes in regions with less anthropogenic activity and substantially lower nutrient concentrations. Consequently, we argue that the conclusions drawn by Bachmann et al. (2013) at the national scale are based upon a statistically flawed analysis. Historical water quality trends used in Bachmann et al. (2013) were inferred from diatom species composition in samples from the top and bottom of sediment cores (USEPA 2010). Because the sediment cores were not dated, a conservative set of criteria was applied to determine whether the bottom of the core represented conditions prior to European settlement. Cores ‘‘were accepted for analysis if they occurred in nutrient ecoregions 2 and 8 where sedimentation rates are known to be relatively low, based on previous studies; lakes with undisturbed, or relatively undisturbed watersheds, and at least moderately long cores for the region; lakes in the Northeast US greater than 25 cm in length were generally considered sufficiently long [...]’’ (USEPA 2010). On the other hand, lakes ‘‘were rejected if: cores less than 20 cm in length, except a few reference lakes that seemed clearly undisturbed [...]; all lakes in nutrient ecoregion 6 with percent watershed disturbance (usually Ag) greater than 50% [...]; and all cores in this ecoregion, regardless of percent watershed disturbance, that were less than 30 cm long were not considered for analysis’’ (USEPA 2010). A number of cores were also classified as ‘‘Uncertain.’’ The analysis of longterm change in nutrient concentrations of natural lakes presented by Bachmann et al. relies on cores from the first category, for which the USEPA was relatively certain that the bottom of the core represented pre-European settlement conditions (Bachmann et al. 2013, p. 946). Probability surveys, such as the NLA, are designed to assess a survey sample of a target population in an unbiased manner (Peck et al. 2013). These designs differ from a simple random subsample by having variable inclusion probabilities (i.e., all sites do not have the same probability of being part of the sample). In the case of the NLA, for example, lakes in different size classes were assigned different probabilities of inclusion to avoid having the smallest size class (4–10 ha) dominate the sample. Population-level estimates, therefore, can be made only if the data analyses make proper use of sample weights, yet Bachmann et al. failed to do this. Further, their use of only a subset of the data, as detailed below, precluded appropriate application of sample weights. Whereas this alone suggests that their conclusions about all natural lakes in the conterminous United States were not drawn in a statistically valid manner, there are several more substantial sources of bias in their analysis. Therefore, their exclusion of sample weights will not be considered further in this comment (i.e., our analysis will also be based on an unweighted analysis of the sampled lakes). In addition to probability sites, which have weights, NLA used data from a set of hand-selected lakes, which do not have weights (they are not part of the probability design). These lakes are hand-chosen because they are likely to be low in nutrients, and therefore these should not be combined with probability data to draw conclusions. Bachmann et al. purportedly analyzed the probability lakes and reference lakes separately. However, a careful analysis of the data used by Bachmann et al. reveals that they incorrectly sorted data into these two categories, resulting in the exclusion of 34 probability lakes from the random sample of natural lakes with good presettlement core data and inclusion of 40 nonrandomly selected lakes (i.e, handpicked potential reference lakes) in their ‘‘probability’’ data set. If the data are correctly sorted, there are 234 probability lakes with valid cores and inferred nutrient concentrations, not the 240 reported by Bachmann et al. (the 234 correct lakes not being a simple subset of the 240 Bachmann et al. lakes). In the following analysis, therefore, the data used by Bachmann et al. and the ‘‘corrected’’ version of the data will be considered separately. The application of the criteria quoted above to determine core confidence necessitated the systematic omission of lakes that have been most subjected to watershed disturbance (i.e., those lakes most likely to be affected by cultural eutrophication) from the Bachmann et al. data. Based on the (unweighted) NLA probabilistic dataset, the majority of natural lakes in the United States are located in the Western Forested Mountains (nutrient ecoregion 2), the South Central Cultivated Great Plains (5), the Corn Belt and Northern Great Plains (6), the Mostly Glaciated Dairy Region (7), and the Nutrient Poor Largely Glaciated Upper Midwest and Northeast (8; Figs. 1, 2). However, the NLA sediment core data used by Bachmann et al. (corrected to represent only lakes in the probability Limnol. Oceanogr., 59(6), 2014, 2226–2230 E 2014, by the Association for the Sciences of Limnology and Oceanography, Inc. doi:10.4319/lo.2014.59.6.2226